University of Alberta NEW REPRESENTATIONS AND APPROXIMATIONS FOR SEQUENTIAL DECISION MAKING UNDER UNCERTAINTY

This dissertation research addresses the challenge of scaling up algorithms for sequential decision making under uncertainty. In my dissertation, I developed new approximation strategies for planning and learning in the presence of uncertainty while maintaining useful theoretical properties that allow larger problems to be tackled than is practical with exact methods. In particular, my research tackles three outstanding issues in sequential decision making in uncertain environments: performing stable generalization during off-policy updates, balancing exploration with exploitation, and handling partial observability of the environment. The first key contribution of my thesis is the development of novel dual representations and algorithms for planning and learning in stochastic environments. This dual view I have developed offers a coherent and comprehensive approach to optimal sequential decision making problems, provides an alternative to standard value function based techniques, and opens new avenues for solving sequential decision making problems. In particular, I have shown that dual dynamic programming algorithms can avoid the divergence problems associated with the standard primal approach, even in the presence of approximation and off-policy updates. Another key contribution of my thesis is the development of a practical action selection strategy that addresses the well known exploration versus exploitation tradeoff in reinforcement learning. The idea is to exploit information in a Bayesian posterior to make intelligent actions by growing an adaptive, sparse lookahead tree. This technique evaluates actions while taking into account any effects they might have on future knowledge, as well as future reward, and outperforms current selection strategies. Finally my thesis also develops a new approach to approximate planning in partially observable Markov decision processes. Here the challenge is to overcome the exponential space required by standard value iteration. For this problem, I introduced a new, quadratic upper bound approximation that can be optimized by semidefinite programming. This approach achieves competitive approximation quality while maintaining a compact representation; requiring computational time and space that is only linear in the number of decisions. Overall, my dissertation research developed new tools for computing optimal sequential decision strategies in stochastic environments, and has contributed significant progress on three key challenges in reinforcement learning.